{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9867604a",
   "metadata": {
    "origin_pos": 1
   },
   "source": [
    "# Gaussian Process Priors\n",
    "\n",
    "Understanding Gaussian processes (GPs) is important for reasoning about model construction and generalization, and for achieving state-of-the-art performance in a variety of applications, including active learning, and hyperparameter tuning in deep learning. GPs are everywhere, and it is in our interests to know what they are and how we can use them.\n",
    "\n",
    "In this section, we introduce Gaussian process _priors_ over functions. In the next notebook, we show how to use these priors to do _posterior inference_ and make predictions. The next section can be viewed as \"GPs in a nutshell\", quickly giving what you need to apply Gaussian processes in practice.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2fddbe46",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:43:58.581002Z",
     "iopub.status.busy": "2023-08-18T19:43:58.580704Z",
     "iopub.status.idle": "2023-08-18T19:44:01.629363Z",
     "shell.execute_reply": "2023-08-18T19:44:01.628441Z"
    },
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.spatial import distance_matrix\n",
    "from d2l import torch as d2l\n",
    "\n",
    "d2l.set_figsize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b57d76",
   "metadata": {
    "origin_pos": 3
   },
   "source": [
    "## Definition\n",
    "\n",
    "A Gaussian process is defined as _a collection of random variables, any finite number of which have a joint Gaussian distribution_. If a function $f(x)$ is a Gaussian process, with _mean function_ $m(x)$ and _covariance function_ or _kernel_ $k(x,x')$, $f(x) \\sim \\mathcal{GP}(m, k)$, then any collection of function values queried at any collection of input points $x$ (times, spatial locations, image pixels, etc.), has a joint multivariate Gaussian distribution with mean vector $\\mu$ and covariance matrix $K$: $f(x_1),\\dots,f(x_n) \\sim \\mathcal{N}(\\mu, K)$, where $\\mu_i = E[f(x_i)] = m(x_i)$ and $K_{ij} = \\textrm{Cov}(f(x_i),f(x_j)) = k(x_i,x_j)$.\n",
    "\n",
    "This definition may seem abstract and inaccessible, but Gaussian processes are in fact very simple objects. Any function\n",
    "\n",
    "$$f(x) = w^{\\top} \\phi(x) = \\langle w, \\phi(x) \\rangle,$$ :eqlabel:`eq_gp-function`\n",
    "\n",
    "with $w$ drawn from a Gaussian (normal) distribution, and $\\phi$ being any vector of basis functions, for example $\\phi(x) = (1, x, x^2, ..., x^d)^{\\top}$,\n",
    "is a Gaussian process. Moreover, any Gaussian process f(x) can be expressed in the form of equation :eqref:`eq_gp-function`. Let's consider a few concrete examples, to begin getting acquainted with Gaussian processes, after which we can appreciate how simple and useful they really are.\n",
    "\n",
    "## A Simple Gaussian Process\n",
    "\n",
    "Suppose $f(x) = w_0 + w_1 x$, and $w_0, w_1 \\sim \\mathcal{N}(0,1)$, with $w_0, w_1, x$ all in one dimension. We can equivalently write this function as the inner product $f(x) = (w_0, w_1)(1, x)^{\\top}$. In :eqref:`eq_gp-function` above, $w = (w_0, w_1)^{\\top}$ and $\\phi(x) = (1,x)^{\\top}$.\n",
    "\n",
    "For any $x$, $f(x)$ is a sum of two Gaussian random variables. Since Gaussians are closed under addition, $f(x)$ is also a Gaussian random variable for any $x$. In fact, we can compute for any particular $x$ that $f(x)$ is $\\mathcal{N}(0,1+x^2)$. Similarly, the joint distribution for any collection of function values, $(f(x_1),\\dots,f(x_n))$, for any collection of inputs $x_1,\\dots,x_n$, is a multivariate Gaussian distribution. Therefore $f(x)$ is a Gaussian process.\n",
    "\n",
    "In short, $f(x)$ is a _random function_, or a _distribution over functions_. We can gain some insights into this distribution by repeatedly sampling values for $w_0, w_1$, and visualizing the corresponding functions $f(x)$, which are straight lines with slopes and different intercepts, as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4c1d5832",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:44:01.633701Z",
     "iopub.status.busy": "2023-08-18T19:44:01.632985Z",
     "iopub.status.idle": "2023-08-18T19:44:01.812038Z",
     "shell.execute_reply": "2023-08-18T19:44:01.810780Z"
    },
    "origin_pos": 4,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"261.160937pt\" height=\"193.034375pt\" viewBox=\"0 0 261.160937 193.034375\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T19:44:01.771268</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 193.034375 \n",
       "L 261.160937 193.034375 \n",
       "L 261.160937 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 58.660938 145.8 \n",
       "L 253.960938 145.8 \n",
       "L 253.960938 7.2 \n",
       "L 58.660938 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"PolyCollection_1\">\n",
       "    <path d=\"M 67.53821 13.5 \n",
       "L 67.53821 139.5 \n",
       "L 71.161587 137.029493 \n",
       "L 74.784963 134.563369 \n",
       "L 78.40834 132.102212 \n",
       "L 82.031717 129.646712 \n",
       "L 85.655093 127.197691 \n",
       "L 89.27847 124.756136 \n",
       "L 92.901847 122.323239 \n",
       "L 96.525223 119.900457 \n",
       "L 100.1486 117.489584 \n",
       "L 103.771976 115.092851 \n",
       "L 107.395353 112.713066 \n",
       "L 111.01873 110.353804 \n",
       "L 114.642106 108.019673 \n",
       "L 118.265483 105.716697 \n",
       "L 121.88886 103.452863 \n",
       "L 125.512236 101.238919 \n",
       "L 129.135613 99.089539 \n",
       "L 132.758989 97.025016 \n",
       "L 136.382366 95.073669 \n",
       "L 140.005743 93.275038 \n",
       "L 143.629119 91.683493 \n",
       "L 147.252496 90.370504 \n",
       "L 150.875873 89.421269 \n",
       "L 154.499249 88.919474 \n",
       "L 158.122626 88.919474 \n",
       "L 161.746002 89.421269 \n",
       "L 165.369379 90.370504 \n",
       "L 168.992756 91.683493 \n",
       "L 172.616132 93.275038 \n",
       "L 176.239509 95.073669 \n",
       "L 179.862886 97.025016 \n",
       "L 183.486262 99.089539 \n",
       "L 187.109639 101.238919 \n",
       "L 190.733015 103.452863 \n",
       "L 194.356392 105.716697 \n",
       "L 197.979769 108.019673 \n",
       "L 201.603145 110.353804 \n",
       "L 205.226522 112.713066 \n",
       "L 208.849899 115.092851 \n",
       "L 212.473275 117.489584 \n",
       "L 216.096652 119.900457 \n",
       "L 219.720028 122.323239 \n",
       "L 223.343405 124.756136 \n",
       "L 226.966782 127.197691 \n",
       "L 230.590158 129.646712 \n",
       "L 234.213535 132.102212 \n",
       "L 237.836912 134.563369 \n",
       "L 241.460288 137.029493 \n",
       "L 245.083665 139.5 \n",
       "L 245.083665 13.5 \n",
       "L 245.083665 13.5 \n",
       "L 241.460288 15.970507 \n",
       "L 237.836912 18.436631 \n",
       "L 234.213535 20.897788 \n",
       "L 230.590158 23.353288 \n",
       "L 226.966782 25.802309 \n",
       "L 223.343405 28.243864 \n",
       "L 219.720028 30.676761 \n",
       "L 216.096652 33.099543 \n",
       "L 212.473275 35.510416 \n",
       "L 208.849899 37.907149 \n",
       "L 205.226522 40.286934 \n",
       "L 201.603145 42.646196 \n",
       "L 197.979769 44.980327 \n",
       "L 194.356392 47.283303 \n",
       "L 190.733015 49.547137 \n",
       "L 187.109639 51.761081 \n",
       "L 183.486262 53.910461 \n",
       "L 179.862886 55.974984 \n",
       "L 176.239509 57.926331 \n",
       "L 172.616132 59.724962 \n",
       "L 168.992756 61.316507 \n",
       "L 165.369379 62.629496 \n",
       "L 161.746002 63.578731 \n",
       "L 158.122626 64.080526 \n",
       "L 154.499249 64.080526 \n",
       "L 150.875873 63.578731 \n",
       "L 147.252496 62.629496 \n",
       "L 143.629119 61.316507 \n",
       "L 140.005743 59.724962 \n",
       "L 136.382366 57.926331 \n",
       "L 132.758989 55.974984 \n",
       "L 129.135613 53.910461 \n",
       "L 125.512236 51.761081 \n",
       "L 121.88886 49.547137 \n",
       "L 118.265483 47.283303 \n",
       "L 114.642106 44.980327 \n",
       "L 111.01873 42.646196 \n",
       "L 107.395353 40.286934 \n",
       "L 103.771976 37.907149 \n",
       "L 100.1486 35.510416 \n",
       "L 96.525223 33.099543 \n",
       "L 92.901847 30.676761 \n",
       "L 89.27847 28.243864 \n",
       "L 85.655093 25.802309 \n",
       "L 82.031717 23.353288 \n",
       "L 78.40834 20.897788 \n",
       "L 74.784963 18.436631 \n",
       "L 71.161587 15.970507 \n",
       "L 67.53821 13.5 \n",
       "z\n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: #1f77b4; fill-opacity: 0.25\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mfb22e49a4d\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfb22e49a4d\" x=\"85.292756\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <g transform=\"translate(77.921662 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfb22e49a4d\" x=\"120.801847\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(113.430753 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfb22e49a4d\" x=\"156.310938\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(153.129688 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfb22e49a4d\" x=\"191.820028\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(188.638778 160.398438) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfb22e49a4d\" x=\"227.329119\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(224.147869 160.398438) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(150.392188 181.675) scale(0.2 -0.2)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path id=\"m623f8eeb2c\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m623f8eeb2c\" x=\"58.660938\" y=\"138.276583\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(30.55625 142.075801) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m623f8eeb2c\" x=\"58.660938\" y=\"107.388291\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(36.91875 111.18751) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m623f8eeb2c\" x=\"58.660938\" y=\"76.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(45.298438 80.299219) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m623f8eeb2c\" x=\"58.660938\" y=\"45.611709\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(45.298438 49.410927) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m623f8eeb2c\" x=\"58.660938\" y=\"14.723417\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(38.935938 18.522636) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(22.396875 93.742188) rotate(-90) scale(0.2 -0.2)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_11\">\n",
       "    <path d=\"M 67.53821 76.5 \n",
       "L 71.161587 76.5 \n",
       "L 74.784963 76.5 \n",
       "L 78.40834 76.5 \n",
       "L 82.031717 76.5 \n",
       "L 85.655093 76.5 \n",
       "L 89.27847 76.5 \n",
       "L 92.901847 76.5 \n",
       "L 96.525223 76.5 \n",
       "L 100.1486 76.5 \n",
       "L 103.771976 76.5 \n",
       "L 107.395353 76.5 \n",
       "L 111.01873 76.5 \n",
       "L 114.642106 76.5 \n",
       "L 118.265483 76.5 \n",
       "L 121.88886 76.5 \n",
       "L 125.512236 76.5 \n",
       "L 129.135613 76.5 \n",
       "L 132.758989 76.5 \n",
       "L 136.382366 76.5 \n",
       "L 140.005743 76.5 \n",
       "L 143.629119 76.5 \n",
       "L 147.252496 76.5 \n",
       "L 150.875873 76.5 \n",
       "L 154.499249 76.5 \n",
       "L 158.122626 76.5 \n",
       "L 161.746002 76.5 \n",
       "L 165.369379 76.5 \n",
       "L 168.992756 76.5 \n",
       "L 172.616132 76.5 \n",
       "L 176.239509 76.5 \n",
       "L 179.862886 76.5 \n",
       "L 183.486262 76.5 \n",
       "L 187.109639 76.5 \n",
       "L 190.733015 76.5 \n",
       "L 194.356392 76.5 \n",
       "L 197.979769 76.5 \n",
       "L 201.603145 76.5 \n",
       "L 205.226522 76.5 \n",
       "L 208.849899 76.5 \n",
       "L 212.473275 76.5 \n",
       "L 216.096652 76.5 \n",
       "L 219.720028 76.5 \n",
       "L 223.343405 76.5 \n",
       "L 226.966782 76.5 \n",
       "L 230.590158 76.5 \n",
       "L 234.213535 76.5 \n",
       "L 237.836912 76.5 \n",
       "L 241.460288 76.5 \n",
       "L 245.083665 76.5 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #000000; stroke-width: 4; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path d=\"M 67.53821 63.124819 \n",
       "L 71.161587 63.518325 \n",
       "L 74.784963 63.911831 \n",
       "L 78.40834 64.305338 \n",
       "L 82.031717 64.698844 \n",
       "L 85.655093 65.09235 \n",
       "L 89.27847 65.485856 \n",
       "L 92.901847 65.879362 \n",
       "L 96.525223 66.272868 \n",
       "L 100.1486 66.666375 \n",
       "L 103.771976 67.059881 \n",
       "L 107.395353 67.453387 \n",
       "L 111.01873 67.846893 \n",
       "L 114.642106 68.240399 \n",
       "L 118.265483 68.633905 \n",
       "L 121.88886 69.027411 \n",
       "L 125.512236 69.420918 \n",
       "L 129.135613 69.814424 \n",
       "L 132.758989 70.20793 \n",
       "L 136.382366 70.601436 \n",
       "L 140.005743 70.994942 \n",
       "L 143.629119 71.388448 \n",
       "L 147.252496 71.781954 \n",
       "L 150.875873 72.175461 \n",
       "L 154.499249 72.568967 \n",
       "L 158.122626 72.962473 \n",
       "L 161.746002 73.355979 \n",
       "L 165.369379 73.749485 \n",
       "L 168.992756 74.142991 \n",
       "L 172.616132 74.536498 \n",
       "L 176.239509 74.930004 \n",
       "L 179.862886 75.32351 \n",
       "L 183.486262 75.717016 \n",
       "L 187.109639 76.110522 \n",
       "L 190.733015 76.504028 \n",
       "L 194.356392 76.897534 \n",
       "L 197.979769 77.291041 \n",
       "L 201.603145 77.684547 \n",
       "L 205.226522 78.078053 \n",
       "L 208.849899 78.471559 \n",
       "L 212.473275 78.865065 \n",
       "L 216.096652 79.258571 \n",
       "L 219.720028 79.652078 \n",
       "L 223.343405 80.045584 \n",
       "L 226.966782 80.43909 \n",
       "L 230.590158 80.832596 \n",
       "L 234.213535 81.226102 \n",
       "L 237.836912 81.619608 \n",
       "L 241.460288 82.013114 \n",
       "L 245.083665 82.406621 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_13\">\n",
       "    <path d=\"M 67.53821 123.00843 \n",
       "L 71.161587 121.36768 \n",
       "L 74.784963 119.726929 \n",
       "L 78.40834 118.086178 \n",
       "L 82.031717 116.445427 \n",
       "L 85.655093 114.804677 \n",
       "L 89.27847 113.163926 \n",
       "L 92.901847 111.523175 \n",
       "L 96.525223 109.882425 \n",
       "L 100.1486 108.241674 \n",
       "L 103.771976 106.600923 \n",
       "L 107.395353 104.960172 \n",
       "L 111.01873 103.319422 \n",
       "L 114.642106 101.678671 \n",
       "L 118.265483 100.03792 \n",
       "L 121.88886 98.39717 \n",
       "L 125.512236 96.756419 \n",
       "L 129.135613 95.115668 \n",
       "L 132.758989 93.474917 \n",
       "L 136.382366 91.834167 \n",
       "L 140.005743 90.193416 \n",
       "L 143.629119 88.552665 \n",
       "L 147.252496 86.911915 \n",
       "L 150.875873 85.271164 \n",
       "L 154.499249 83.630413 \n",
       "L 158.122626 81.989662 \n",
       "L 161.746002 80.348912 \n",
       "L 165.369379 78.708161 \n",
       "L 168.992756 77.06741 \n",
       "L 172.616132 75.42666 \n",
       "L 176.239509 73.785909 \n",
       "L 179.862886 72.145158 \n",
       "L 183.486262 70.504407 \n",
       "L 187.109639 68.863657 \n",
       "L 190.733015 67.222906 \n",
       "L 194.356392 65.582155 \n",
       "L 197.979769 63.941404 \n",
       "L 201.603145 62.300654 \n",
       "L 205.226522 60.659903 \n",
       "L 208.849899 59.019152 \n",
       "L 212.473275 57.378402 \n",
       "L 216.096652 55.737651 \n",
       "L 219.720028 54.0969 \n",
       "L 223.343405 52.456149 \n",
       "L 226.966782 50.815399 \n",
       "L 230.590158 49.174648 \n",
       "L 234.213535 47.533897 \n",
       "L 237.836912 45.893147 \n",
       "L 241.460288 44.252396 \n",
       "L 245.083665 42.611645 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_14\">\n",
       "    <path d=\"M 67.53821 93.522973 \n",
       "L 71.161587 93.2744 \n",
       "L 74.784963 93.025826 \n",
       "L 78.40834 92.777252 \n",
       "L 82.031717 92.528679 \n",
       "L 85.655093 92.280105 \n",
       "L 89.27847 92.031532 \n",
       "L 92.901847 91.782958 \n",
       "L 96.525223 91.534385 \n",
       "L 100.1486 91.285811 \n",
       "L 103.771976 91.037238 \n",
       "L 107.395353 90.788664 \n",
       "L 111.01873 90.540091 \n",
       "L 114.642106 90.291517 \n",
       "L 118.265483 90.042944 \n",
       "L 121.88886 89.79437 \n",
       "L 125.512236 89.545797 \n",
       "L 129.135613 89.297223 \n",
       "L 132.758989 89.04865 \n",
       "L 136.382366 88.800076 \n",
       "L 140.005743 88.551503 \n",
       "L 143.629119 88.302929 \n",
       "L 147.252496 88.054355 \n",
       "L 150.875873 87.805782 \n",
       "L 154.499249 87.557208 \n",
       "L 158.122626 87.308635 \n",
       "L 161.746002 87.060061 \n",
       "L 165.369379 86.811488 \n",
       "L 168.992756 86.562914 \n",
       "L 172.616132 86.314341 \n",
       "L 176.239509 86.065767 \n",
       "L 179.862886 85.817194 \n",
       "L 183.486262 85.56862 \n",
       "L 187.109639 85.320047 \n",
       "L 190.733015 85.071473 \n",
       "L 194.356392 84.8229 \n",
       "L 197.979769 84.574326 \n",
       "L 201.603145 84.325753 \n",
       "L 205.226522 84.077179 \n",
       "L 208.849899 83.828606 \n",
       "L 212.473275 83.580032 \n",
       "L 216.096652 83.331458 \n",
       "L 219.720028 83.082885 \n",
       "L 223.343405 82.834311 \n",
       "L 226.966782 82.585738 \n",
       "L 230.590158 82.337164 \n",
       "L 234.213535 82.088591 \n",
       "L 237.836912 81.840017 \n",
       "L 241.460288 81.591444 \n",
       "L 245.083665 81.34287 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #2ca02c; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_15\">\n",
       "    <path d=\"M 67.53821 119.99885 \n",
       "L 71.161587 118.278119 \n",
       "L 74.784963 116.557388 \n",
       "L 78.40834 114.836657 \n",
       "L 82.031717 113.115926 \n",
       "L 85.655093 111.395194 \n",
       "L 89.27847 109.674463 \n",
       "L 92.901847 107.953732 \n",
       "L 96.525223 106.233001 \n",
       "L 100.1486 104.51227 \n",
       "L 103.771976 102.791539 \n",
       "L 107.395353 101.070808 \n",
       "L 111.01873 99.350077 \n",
       "L 114.642106 97.629346 \n",
       "L 118.265483 95.908614 \n",
       "L 121.88886 94.187883 \n",
       "L 125.512236 92.467152 \n",
       "L 129.135613 90.746421 \n",
       "L 132.758989 89.02569 \n",
       "L 136.382366 87.304959 \n",
       "L 140.005743 85.584228 \n",
       "L 143.629119 83.863497 \n",
       "L 147.252496 82.142766 \n",
       "L 150.875873 80.422034 \n",
       "L 154.499249 78.701303 \n",
       "L 158.122626 76.980572 \n",
       "L 161.746002 75.259841 \n",
       "L 165.369379 73.53911 \n",
       "L 168.992756 71.818379 \n",
       "L 172.616132 70.097648 \n",
       "L 176.239509 68.376917 \n",
       "L 179.862886 66.656186 \n",
       "L 183.486262 64.935454 \n",
       "L 187.109639 63.214723 \n",
       "L 190.733015 61.493992 \n",
       "L 194.356392 59.773261 \n",
       "L 197.979769 58.05253 \n",
       "L 201.603145 56.331799 \n",
       "L 205.226522 54.611068 \n",
       "L 208.849899 52.890337 \n",
       "L 212.473275 51.169606 \n",
       "L 216.096652 49.448874 \n",
       "L 219.720028 47.728143 \n",
       "L 223.343405 46.007412 \n",
       "L 226.966782 44.286681 \n",
       "L 230.590158 42.56595 \n",
       "L 234.213535 40.845219 \n",
       "L 237.836912 39.124488 \n",
       "L 241.460288 37.403757 \n",
       "L 245.083665 35.683026 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #d62728; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_16\">\n",
       "    <path d=\"M 67.53821 81.367696 \n",
       "L 71.161587 80.639364 \n",
       "L 74.784963 79.911031 \n",
       "L 78.40834 79.182699 \n",
       "L 82.031717 78.454367 \n",
       "L 85.655093 77.726034 \n",
       "L 89.27847 76.997702 \n",
       "L 92.901847 76.269369 \n",
       "L 96.525223 75.541037 \n",
       "L 100.1486 74.812704 \n",
       "L 103.771976 74.084372 \n",
       "L 107.395353 73.35604 \n",
       "L 111.01873 72.627707 \n",
       "L 114.642106 71.899375 \n",
       "L 118.265483 71.171042 \n",
       "L 121.88886 70.44271 \n",
       "L 125.512236 69.714377 \n",
       "L 129.135613 68.986045 \n",
       "L 132.758989 68.257713 \n",
       "L 136.382366 67.52938 \n",
       "L 140.005743 66.801048 \n",
       "L 143.629119 66.072715 \n",
       "L 147.252496 65.344383 \n",
       "L 150.875873 64.61605 \n",
       "L 154.499249 63.887718 \n",
       "L 158.122626 63.159386 \n",
       "L 161.746002 62.431053 \n",
       "L 165.369379 61.702721 \n",
       "L 168.992756 60.974388 \n",
       "L 172.616132 60.246056 \n",
       "L 176.239509 59.517723 \n",
       "L 179.862886 58.789391 \n",
       "L 183.486262 58.061059 \n",
       "L 187.109639 57.332726 \n",
       "L 190.733015 56.604394 \n",
       "L 194.356392 55.876061 \n",
       "L 197.979769 55.147729 \n",
       "L 201.603145 54.419396 \n",
       "L 205.226522 53.691064 \n",
       "L 208.849899 52.962731 \n",
       "L 212.473275 52.234399 \n",
       "L 216.096652 51.506067 \n",
       "L 219.720028 50.777734 \n",
       "L 223.343405 50.049402 \n",
       "L 226.966782 49.321069 \n",
       "L 230.590158 48.592737 \n",
       "L 234.213535 47.864404 \n",
       "L 237.836912 47.136072 \n",
       "L 241.460288 46.40774 \n",
       "L 245.083665 45.679407 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #9467bd; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 67.53821 94.496441 \n",
       "L 71.161587 93.624341 \n",
       "L 74.784963 92.752241 \n",
       "L 78.40834 91.880142 \n",
       "L 82.031717 91.008042 \n",
       "L 85.655093 90.135942 \n",
       "L 89.27847 89.263842 \n",
       "L 92.901847 88.391742 \n",
       "L 96.525223 87.519642 \n",
       "L 100.1486 86.647542 \n",
       "L 103.771976 85.775443 \n",
       "L 107.395353 84.903343 \n",
       "L 111.01873 84.031243 \n",
       "L 114.642106 83.159143 \n",
       "L 118.265483 82.287043 \n",
       "L 121.88886 81.414943 \n",
       "L 125.512236 80.542843 \n",
       "L 129.135613 79.670744 \n",
       "L 132.758989 78.798644 \n",
       "L 136.382366 77.926544 \n",
       "L 140.005743 77.054444 \n",
       "L 143.629119 76.182344 \n",
       "L 147.252496 75.310244 \n",
       "L 150.875873 74.438144 \n",
       "L 154.499249 73.566044 \n",
       "L 158.122626 72.693945 \n",
       "L 161.746002 71.821845 \n",
       "L 165.369379 70.949745 \n",
       "L 168.992756 70.077645 \n",
       "L 172.616132 69.205545 \n",
       "L 176.239509 68.333445 \n",
       "L 179.862886 67.461345 \n",
       "L 183.486262 66.589246 \n",
       "L 187.109639 65.717146 \n",
       "L 190.733015 64.845046 \n",
       "L 194.356392 63.972946 \n",
       "L 197.979769 63.100846 \n",
       "L 201.603145 62.228746 \n",
       "L 205.226522 61.356646 \n",
       "L 208.849899 60.484547 \n",
       "L 212.473275 59.612447 \n",
       "L 216.096652 58.740347 \n",
       "L 219.720028 57.868247 \n",
       "L 223.343405 56.996147 \n",
       "L 226.966782 56.124047 \n",
       "L 230.590158 55.251947 \n",
       "L 234.213535 54.379847 \n",
       "L 237.836912 53.507748 \n",
       "L 241.460288 52.635648 \n",
       "L 245.083665 51.763548 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #8c564b; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 67.53821 64.032766 \n",
       "L 71.161587 64.143852 \n",
       "L 74.784963 64.254938 \n",
       "L 78.40834 64.366024 \n",
       "L 82.031717 64.47711 \n",
       "L 85.655093 64.588197 \n",
       "L 89.27847 64.699283 \n",
       "L 92.901847 64.810369 \n",
       "L 96.525223 64.921455 \n",
       "L 100.1486 65.032541 \n",
       "L 103.771976 65.143627 \n",
       "L 107.395353 65.254714 \n",
       "L 111.01873 65.3658 \n",
       "L 114.642106 65.476886 \n",
       "L 118.265483 65.587972 \n",
       "L 121.88886 65.699058 \n",
       "L 125.512236 65.810145 \n",
       "L 129.135613 65.921231 \n",
       "L 132.758989 66.032317 \n",
       "L 136.382366 66.143403 \n",
       "L 140.005743 66.254489 \n",
       "L 143.629119 66.365576 \n",
       "L 147.252496 66.476662 \n",
       "L 150.875873 66.587748 \n",
       "L 154.499249 66.698834 \n",
       "L 158.122626 66.80992 \n",
       "L 161.746002 66.921007 \n",
       "L 165.369379 67.032093 \n",
       "L 168.992756 67.143179 \n",
       "L 172.616132 67.254265 \n",
       "L 176.239509 67.365351 \n",
       "L 179.862886 67.476437 \n",
       "L 183.486262 67.587524 \n",
       "L 187.109639 67.69861 \n",
       "L 190.733015 67.809696 \n",
       "L 194.356392 67.920782 \n",
       "L 197.979769 68.031868 \n",
       "L 201.603145 68.142955 \n",
       "L 205.226522 68.254041 \n",
       "L 208.849899 68.365127 \n",
       "L 212.473275 68.476213 \n",
       "L 216.096652 68.587299 \n",
       "L 219.720028 68.698386 \n",
       "L 223.343405 68.809472 \n",
       "L 226.966782 68.920558 \n",
       "L 230.590158 69.031644 \n",
       "L 234.213535 69.14273 \n",
       "L 237.836912 69.253817 \n",
       "L 241.460288 69.364903 \n",
       "L 245.083665 69.475989 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #e377c2; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_19\">\n",
       "    <path d=\"M 67.53821 62.399207 \n",
       "L 71.161587 62.756201 \n",
       "L 74.784963 63.113195 \n",
       "L 78.40834 63.470189 \n",
       "L 82.031717 63.827183 \n",
       "L 85.655093 64.184176 \n",
       "L 89.27847 64.54117 \n",
       "L 92.901847 64.898164 \n",
       "L 96.525223 65.255158 \n",
       "L 100.1486 65.612151 \n",
       "L 103.771976 65.969145 \n",
       "L 107.395353 66.326139 \n",
       "L 111.01873 66.683133 \n",
       "L 114.642106 67.040127 \n",
       "L 118.265483 67.39712 \n",
       "L 121.88886 67.754114 \n",
       "L 125.512236 68.111108 \n",
       "L 129.135613 68.468102 \n",
       "L 132.758989 68.825096 \n",
       "L 136.382366 69.182089 \n",
       "L 140.005743 69.539083 \n",
       "L 143.629119 69.896077 \n",
       "L 147.252496 70.253071 \n",
       "L 150.875873 70.610064 \n",
       "L 154.499249 70.967058 \n",
       "L 158.122626 71.324052 \n",
       "L 161.746002 71.681046 \n",
       "L 165.369379 72.03804 \n",
       "L 168.992756 72.395033 \n",
       "L 172.616132 72.752027 \n",
       "L 176.239509 73.109021 \n",
       "L 179.862886 73.466015 \n",
       "L 183.486262 73.823008 \n",
       "L 187.109639 74.180002 \n",
       "L 190.733015 74.536996 \n",
       "L 194.356392 74.89399 \n",
       "L 197.979769 75.250984 \n",
       "L 201.603145 75.607977 \n",
       "L 205.226522 75.964971 \n",
       "L 208.849899 76.321965 \n",
       "L 212.473275 76.678959 \n",
       "L 216.096652 77.035953 \n",
       "L 219.720028 77.392946 \n",
       "L 223.343405 77.74994 \n",
       "L 226.966782 78.106934 \n",
       "L 230.590158 78.463928 \n",
       "L 234.213535 78.820921 \n",
       "L 237.836912 79.177915 \n",
       "L 241.460288 79.534909 \n",
       "L 245.083665 79.891903 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #7f7f7f; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_20\">\n",
       "    <path d=\"M 67.53821 50.572397 \n",
       "L 71.161587 51.630685 \n",
       "L 74.784963 52.688972 \n",
       "L 78.40834 53.74726 \n",
       "L 82.031717 54.805548 \n",
       "L 85.655093 55.863836 \n",
       "L 89.27847 56.922124 \n",
       "L 92.901847 57.980411 \n",
       "L 96.525223 59.038699 \n",
       "L 100.1486 60.096987 \n",
       "L 103.771976 61.155275 \n",
       "L 107.395353 62.213563 \n",
       "L 111.01873 63.27185 \n",
       "L 114.642106 64.330138 \n",
       "L 118.265483 65.388426 \n",
       "L 121.88886 66.446714 \n",
       "L 125.512236 67.505001 \n",
       "L 129.135613 68.563289 \n",
       "L 132.758989 69.621577 \n",
       "L 136.382366 70.679865 \n",
       "L 140.005743 71.738153 \n",
       "L 143.629119 72.79644 \n",
       "L 147.252496 73.854728 \n",
       "L 150.875873 74.913016 \n",
       "L 154.499249 75.971304 \n",
       "L 158.122626 77.029592 \n",
       "L 161.746002 78.087879 \n",
       "L 165.369379 79.146167 \n",
       "L 168.992756 80.204455 \n",
       "L 172.616132 81.262743 \n",
       "L 176.239509 82.32103 \n",
       "L 179.862886 83.379318 \n",
       "L 183.486262 84.437606 \n",
       "L 187.109639 85.495894 \n",
       "L 190.733015 86.554182 \n",
       "L 194.356392 87.612469 \n",
       "L 197.979769 88.670757 \n",
       "L 201.603145 89.729045 \n",
       "L 205.226522 90.787333 \n",
       "L 208.849899 91.845621 \n",
       "L 212.473275 92.903908 \n",
       "L 216.096652 93.962196 \n",
       "L 219.720028 95.020484 \n",
       "L 223.343405 96.078772 \n",
       "L 226.966782 97.13706 \n",
       "L 230.590158 98.195347 \n",
       "L 234.213535 99.253635 \n",
       "L 237.836912 100.311923 \n",
       "L 241.460288 101.370211 \n",
       "L 245.083665 102.428498 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #bcbd22; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 67.53821 79.12913 \n",
       "L 71.161587 78.744038 \n",
       "L 74.784963 78.358946 \n",
       "L 78.40834 77.973854 \n",
       "L 82.031717 77.588762 \n",
       "L 85.655093 77.203671 \n",
       "L 89.27847 76.818579 \n",
       "L 92.901847 76.433487 \n",
       "L 96.525223 76.048395 \n",
       "L 100.1486 75.663303 \n",
       "L 103.771976 75.278211 \n",
       "L 107.395353 74.893119 \n",
       "L 111.01873 74.508027 \n",
       "L 114.642106 74.122935 \n",
       "L 118.265483 73.737843 \n",
       "L 121.88886 73.352751 \n",
       "L 125.512236 72.967659 \n",
       "L 129.135613 72.582568 \n",
       "L 132.758989 72.197476 \n",
       "L 136.382366 71.812384 \n",
       "L 140.005743 71.427292 \n",
       "L 143.629119 71.0422 \n",
       "L 147.252496 70.657108 \n",
       "L 150.875873 70.272016 \n",
       "L 154.499249 69.886924 \n",
       "L 158.122626 69.501832 \n",
       "L 161.746002 69.11674 \n",
       "L 165.369379 68.731648 \n",
       "L 168.992756 68.346556 \n",
       "L 172.616132 67.961464 \n",
       "L 176.239509 67.576373 \n",
       "L 179.862886 67.191281 \n",
       "L 183.486262 66.806189 \n",
       "L 187.109639 66.421097 \n",
       "L 190.733015 66.036005 \n",
       "L 194.356392 65.650913 \n",
       "L 197.979769 65.265821 \n",
       "L 201.603145 64.880729 \n",
       "L 205.226522 64.495637 \n",
       "L 208.849899 64.110545 \n",
       "L 212.473275 63.725453 \n",
       "L 216.096652 63.340361 \n",
       "L 219.720028 62.95527 \n",
       "L 223.343405 62.570178 \n",
       "L 226.966782 62.185086 \n",
       "L 230.590158 61.799994 \n",
       "L 234.213535 61.414902 \n",
       "L 237.836912 61.02981 \n",
       "L 241.460288 60.644718 \n",
       "L 245.083665 60.259626 \n",
       "\" clip-path=\"url(#p87f53ca897)\" style=\"fill: none; stroke: #17becf; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 58.660938 145.8 \n",
       "L 58.660938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 253.960938 145.8 \n",
       "L 253.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 58.660938 145.8 \n",
       "L 253.960938 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 58.660938 7.2 \n",
       "L 253.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p87f53ca897\">\n",
       "   <rect x=\"58.660938\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def lin_func(x, n_sample):\n",
    "    preds = np.zeros((n_sample, x.shape[0]))\n",
    "    for ii in range(n_sample):\n",
    "        w = np.random.normal(0, 1, 2)\n",
    "        y = w[0] + w[1] * x\n",
    "        preds[ii, :] = y\n",
    "    return preds\n",
    "\n",
    "x_points = np.linspace(-5, 5, 50)\n",
    "outs = lin_func(x_points, 10)\n",
    "lw_bd = -2 * np.sqrt((1 + x_points ** 2))\n",
    "up_bd = 2 * np.sqrt((1 + x_points ** 2))\n",
    "\n",
    "d2l.plt.fill_between(x_points, lw_bd, up_bd, alpha=0.25)\n",
    "d2l.plt.plot(x_points, np.zeros(len(x_points)), linewidth=4, color='black')\n",
    "d2l.plt.plot(x_points, outs.T)\n",
    "d2l.plt.xlabel(\"x\", fontsize=20)\n",
    "d2l.plt.ylabel(\"f(x)\", fontsize=20)\n",
    "d2l.plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a25e08",
   "metadata": {
    "origin_pos": 5
   },
   "source": [
    "If $w_0$ and $w_1$ are instead drawn from $\\mathcal{N}(0,\\alpha^2)$, how do you imagine varying $\\alpha$ affects the distribution over functions?\n",
    "\n",
    "## From Weight Space to Function Space\n",
    "\n",
    "In the plot above, we saw how a distribution over parameters in a model induces a distribution over functions. While we often have ideas about the functions we want to model --- whether they're smooth, periodic, quickly varying, etc. --- it is relatively tedious to reason about the parameters, which are largely uninterpretable. Fortunately, Gaussian processes provide an easy mechanism to reason _directly_ about functions. Since a Gaussian distribution is entirely defined by its first two moments, its mean and covariance matrix, a Gaussian process by extension is defined by its mean function and covariance function.\n",
    "\n",
    "In the above example, the mean function\n",
    "\n",
    "$$m(x) = E[f(x)] = E[w_0 + w_1x] = E[w_0] + E[w_1]x = 0+0 = 0.$$\n",
    "\n",
    "Similarly, the covariance function is\n",
    "\n",
    "$$k(x,x') = \\textrm{Cov}(f(x),f(x')) = E[f(x)f(x')]-E[f(x)]E[f(x')] = E[w_0^2 + w_0w_1x' + w_1w_0x + w_1^2xx'] = 1 + xx'.$$\n",
    "\n",
    "Our distribution over functions can now be directly specified and sampled from, without needing to sample from the distribution over parameters. For example, to draw from $f(x)$, we can simply form our multivariate Gaussian distribution associated with any collection of $x$ we want to query, and sample from it directly. We will begin to see just how advantageous this formulation will be.\n",
    "\n",
    "First, we note that essentially the same derivation for the simple straight line model above can be applied to find the mean and covariance function for _any_ model of the form $f(x) = w^{\\top} \\phi(x)$, with $w \\sim \\mathcal{N}(u,S)$. In this case, the mean function $m(x) = u^{\\top}\\phi(x)$, and the covariance function $k(x,x') = \\phi(x)^{\\top}S\\phi(x')$. Since $\\phi(x)$ can represent a vector of any non-linear basis functions, we are considering a very general model class, including models with an even an _infinite_ number of parameters.\n",
    "\n",
    "## The Radial Basis Function (RBF) Kernel\n",
    "\n",
    "The _radial basis function_ (RBF) kernel is the most popular covariance function for Gaussian processes, and kernel machines in general.\n",
    "This kernel has the form $k_{\\textrm{RBF}}(x,x') = a^2\\exp\\left(-\\frac{1}{2\\ell^2}||x-x'||^2\\right)$, where $a$ is an amplitude parameter, and $\\ell$ is a _lengthscale_ hyperparameter.\n",
    "\n",
    "Let's derive this kernel starting from weight space. Consider the function\n",
    "\n",
    "$$f(x) = \\sum_{i=1}^J w_i \\phi_i(x), w_i  \\sim \\mathcal{N}\\left(0,\\frac{\\sigma^2}{J}\\right), \\phi_i(x) = \\exp\\left(-\\frac{(x-c_i)^2}{2\\ell^2 }\\right).$$\n",
    "\n",
    "$f(x)$ is a sum of radial basis functions, with width $\\ell$, centred at the points $c_i$, as shown in the following figure.\n",
    "\n",
    "\n",
    "We can recognize $f(x)$ as having the form $w^{\\top} \\phi(x)$, where $w = (w_1,\\dots,w_J)^{\\top}$ and $\\phi(x)$ is a vector containing each of the radial basis functions. The covariance function of this Gaussian process is then\n",
    "\n",
    "$$k(x,x') = \\frac{\\sigma^2}{J} \\sum_{i=1}^{J} \\phi_i(x)\\phi_i(x').$$\n",
    "\n",
    "Now let's consider what happens as we take the number of parameters (and basis functions) to infinity. Let $c_J = \\log J$, $c_1 = -\\log J$, and $c_{i+1}-c_{i} = \\Delta c = 2\\frac{\\log J}{J}$, and $J \\to \\infty$. The covariance function becomes the Riemann sum:\n",
    "\n",
    "$$k(x,x') = \\lim_{J \\to \\infty} \\frac{\\sigma^2}{J} \\sum_{i=1}^{J} \\phi_i(x)\\phi_i(x') = \\int_{c_0}^{c_\\infty} \\phi_c(x)\\phi_c(x') dc.$$\n",
    "\n",
    "By setting $c_0 = -\\infty$ and $c_\\infty = \\infty$, we spread the infinitely many basis functions across the whole real line, each\n",
    "a distance $\\Delta c \\to 0$ apart:\n",
    "\n",
    "$$k(x,x') = \\int_{-\\infty}^{\\infty} \\exp(-\\frac{(x-c)^2}{2\\ell^2}) \\exp(-\\frac{(x'-c)^2}{2\\ell^2 }) dc = \\sqrt{\\pi}\\ell \\sigma^2 \\exp(-\\frac{(x-x')^2}{2(\\sqrt{2} \\ell)^2}) \\propto k_{\\textrm{RBF}}(x,x').$$\n",
    "\n",
    "It is worth taking a moment to absorb what we have done here. By moving into the function space representation, we have derived how to represent a model with an _infinite_ number of parameters, using a finite amount of computation. A Gaussian process with an RBF kernel is a _universal approximator_, capable of representing any continuous function to arbitrary precision. We can intuitively see why from the above derivation. We can collapse each radial basis function to a point mass taking $\\ell \\to 0$, and give each point mass any height we wish.\n",
    "\n",
    "So a Gaussian process with an RBF kernel is a model with an infinite number of parameters and much more flexibility than any finite neural network. Perhaps all the fuss about _overparametrized_ neural networks is misplaced. As we will see, GPs with RBF kernels do not overfit, and in fact provide especially compelling generalization performance on small datasets. Moreover, the examples in :cite:`zhang2021understanding`, such as the ability to fit images with random labels perfectly, but still generalize well on structured problems, (can be perfectly reproduced using Gaussian processes) :cite:`wilson2020bayesian`. Neural networks are not as distinct as we make them out to be.\n",
    "\n",
    "We can build further intuition about Gaussian processes with RBF kernels, and hyperparameters such as _length-scale_, by sampling directly from the distribution over functions. As before, this involves a simple procedure:\n",
    "\n",
    "1. Choose the input $x$ points we want to query the GP: $x_1,\\dots,x_n$.\n",
    "2. Evaluate $m(x_i)$, $i = 1,\\dots,n$, and $k(x_i,x_j)$ for $i,j = 1,\\dots,n$ to respectively form the mean vector and covariance matrix $\\mu$ and $K$, where $(f(x_1),\\dots,f(x_n)) \\sim \\mathcal{N}(\\mu, K)$.\n",
    "3. Sample from this multivariate Gaussian distribution to obtain the sample function values.\n",
    "4. Sample more times to visualize more sample functions queried at those points.\n",
    "\n",
    "We illustrate this process in the figure below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "674be2b9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:44:01.816414Z",
     "iopub.status.busy": "2023-08-18T19:44:01.816103Z",
     "iopub.status.idle": "2023-08-18T19:44:02.015532Z",
     "shell.execute_reply": "2023-08-18T19:44:02.014585Z"
    },
    "origin_pos": 6,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"231.442188pt\" height=\"171.103233pt\" viewBox=\"0 0 231.442188 171.103233\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T19:44:01.980516</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 171.103233 \n",
       "L 231.442188 171.103233 \n",
       "L 231.442188 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 28.942188 147.225108 \n",
       "L 224.242188 147.225108 \n",
       "L 224.242188 8.625108 \n",
       "L 28.942188 8.625108 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m257ba2fd9a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"37.81946\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(34.63821 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"73.328551\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(70.147301 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"108.837642\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(105.656392 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"144.346733\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 3 -->\n",
       "      <g transform=\"translate(141.165483 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"179.855824\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(176.674574 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m257ba2fd9a\" x=\"215.364915\" y=\"147.225108\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(212.183665 161.823545) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <defs>\n",
       "       <path id=\"md3393c5bcf\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#md3393c5bcf\" x=\"28.942188\" y=\"119.387539\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(7.2 123.186757) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md3393c5bcf\" x=\"28.942188\" y=\"83.258099\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(15.579688 87.057317) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md3393c5bcf\" x=\"28.942188\" y=\"47.128659\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(15.579688 50.927877) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md3393c5bcf\" x=\"28.942188\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(15.579688 14.798437) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_11\">\n",
       "    <path d=\"M 37.81946 46.136717 \n",
       "L 41.442837 53.653019 \n",
       "L 45.066213 60.004343 \n",
       "L 48.68959 64.935431 \n",
       "L 52.312967 68.2829 \n",
       "L 55.936343 69.995085 \n",
       "L 59.55972 70.1427 \n",
       "L 63.183097 68.918358 \n",
       "L 66.806473 66.624244 \n",
       "L 70.42985 63.648779 \n",
       "L 74.053226 60.434251 \n",
       "L 77.676603 57.438753 \n",
       "L 81.29998 55.096237 \n",
       "L 84.923356 53.778905 \n",
       "L 88.546733 53.765656 \n",
       "L 92.17011 55.219654 \n",
       "L 95.793486 58.176893 \n",
       "L 99.416863 62.546436 \n",
       "L 103.040239 68.121736 \n",
       "L 106.663616 74.601353 \n",
       "L 110.286993 81.616649 \n",
       "L 113.910369 88.763445 \n",
       "L 117.533746 95.634562 \n",
       "L 121.157123 101.850233 \n",
       "L 124.780499 107.083938 \n",
       "L 128.403876 111.081663 \n",
       "L 132.027252 113.673543 \n",
       "L 135.650629 114.777532 \n",
       "L 139.274006 114.39571 \n",
       "L 142.897382 112.604394 \n",
       "L 146.520759 109.539932 \n",
       "L 150.144136 105.382118 \n",
       "L 153.767512 100.337447 \n",
       "L 157.390889 94.623906 \n",
       "L 161.014265 88.458724 \n",
       "L 164.637642 82.049729 \n",
       "L 168.261019 75.590478 \n",
       "L 171.884395 69.258526 \n",
       "L 175.507772 63.21598 \n",
       "L 179.131149 57.611028 \n",
       "L 182.754525 52.579375 \n",
       "L 186.377902 48.244434 \n",
       "L 190.001278 44.715766 \n",
       "L 193.624655 42.085522 \n",
       "L 197.248032 40.423278 \n",
       "L 200.871408 39.769991 \n",
       "L 204.494785 40.132257 \n",
       "L 208.118162 41.478018 \n",
       "L 211.741538 43.734986 \n",
       "L 215.364915 46.792645 \n",
       "\" clip-path=\"url(#p0ebccbb441)\" style=\"fill: none; stroke: #1f77b4; stroke-opacity: 0.5; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path d=\"M 37.81946 49.504199 \n",
       "L 41.442837 47.586298 \n",
       "L 45.066213 45.940146 \n",
       "L 48.68959 44.532133 \n",
       "L 52.312967 43.299444 \n",
       "L 55.936343 42.163307 \n",
       "L 59.55972 41.045501 \n",
       "L 63.183097 39.885827 \n",
       "L 66.806473 38.657942 \n",
       "L 70.42985 37.381083 \n",
       "L 74.053226 36.125736 \n",
       "L 77.676603 35.01218 \n",
       "L 81.29998 34.201913 \n",
       "L 84.923356 33.882878 \n",
       "L 88.546733 34.25048 \n",
       "L 92.17011 35.486612 \n",
       "L 95.793486 37.739346 \n",
       "L 99.416863 41.10548 \n",
       "L 103.040239 45.617715 \n",
       "L 106.663616 51.237554 \n",
       "L 110.286993 57.854203 \n",
       "L 113.910369 65.28921 \n",
       "L 117.533746 73.306074 \n",
       "L 121.157123 81.623814 \n",
       "L 124.780499 89.933377 \n",
       "L 128.403876 97.915846 \n",
       "L 132.027252 105.261493 \n",
       "L 135.650629 111.688723 \n",
       "L 139.274006 116.961979 \n",
       "L 142.897382 120.907635 \n",
       "L 146.520759 123.426589 \n",
       "L 150.144136 124.502433 \n",
       "L 153.767512 124.203906 \n",
       "L 157.390889 122.680783 \n",
       "L 161.014265 120.152758 \n",
       "L 164.637642 116.891767 \n",
       "L 168.261019 113.198978 \n",
       "L 171.884395 109.378724 \n",
       "L 175.507772 105.7122 \n",
       "L 179.131149 102.434121 \n",
       "L 182.754525 99.715319 \n",
       "L 186.377902 97.6535 \n",
       "L 190.001278 96.273174 \n",
       "L 193.624655 95.534383 \n",
       "L 197.248032 95.348402 \n",
       "L 200.871408 95.597323 \n",
       "L 204.494785 96.15413 \n",
       "L 208.118162 96.899617 \n",
       "L 211.741538 97.733465 \n",
       "L 215.364915 98.578058 \n",
       "\" clip-path=\"url(#p0ebccbb441)\" style=\"fill: none; stroke: #ff7f0e; stroke-opacity: 0.5; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_13\">\n",
       "    <path d=\"M 37.81946 77.400652 \n",
       "L 41.442837 72.512938 \n",
       "L 45.066213 68.025249 \n",
       "L 48.68959 63.935239 \n",
       "L 52.312967 60.219782 \n",
       "L 55.936343 56.849318 \n",
       "L 59.55972 53.802111 \n",
       "L 63.183097 51.076371 \n",
       "L 66.806473 48.698612 \n",
       "L 70.42985 46.7269 \n",
       "L 74.053226 45.248199 \n",
       "L 77.676603 44.369913 \n",
       "L 81.29998 44.206382 \n",
       "L 84.923356 44.861918 \n",
       "L 88.546733 46.412683 \n",
       "L 92.17011 48.88998 \n",
       "L 95.793486 52.267646 \n",
       "L 99.416863 56.45581 \n",
       "L 103.040239 61.30256 \n",
       "L 106.663616 66.603787 \n",
       "L 110.286993 72.120503 \n",
       "L 113.910369 77.601451 \n",
       "L 117.533746 82.808129 \n",
       "L 121.157123 87.53874 \n",
       "L 124.780499 91.647734 \n",
       "L 128.403876 95.058164 \n",
       "L 132.027252 97.765201 \n",
       "L 135.650629 99.830493 \n",
       "L 139.274006 101.36831 \n",
       "L 142.897382 102.525767 \n",
       "L 146.520759 103.459997 \n",
       "L 150.144136 104.315629 \n",
       "L 153.767512 105.205575 \n",
       "L 157.390889 106.197534 \n",
       "L 161.014265 107.307595 \n",
       "L 164.637642 108.501248 \n",
       "L 168.261019 109.700962 \n",
       "L 171.884395 110.798647 \n",
       "L 175.507772 111.670726 \n",
       "L 179.131149 112.193427 \n",
       "L 182.754525 112.256046 \n",
       "L 186.377902 111.770543 \n",
       "L 190.001278 110.676594 \n",
       "L 193.624655 108.942142 \n",
       "L 197.248032 106.560154 \n",
       "L 200.871408 103.543117 \n",
       "L 204.494785 99.916919 \n",
       "L 208.118162 95.715777 \n",
       "L 211.741538 90.979589 \n",
       "L 215.364915 85.754391 \n",
       "\" clip-path=\"url(#p0ebccbb441)\" style=\"fill: none; stroke: #2ca02c; stroke-opacity: 0.5; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_14\">\n",
       "    <path d=\"M 37.81946 66.252324 \n",
       "L 41.442837 65.59773 \n",
       "L 45.066213 65.015711 \n",
       "L 48.68959 64.453796 \n",
       "L 52.312967 63.863278 \n",
       "L 55.936343 63.202609 \n",
       "L 59.55972 62.440296 \n",
       "L 63.183097 61.557318 \n",
       "L 66.806473 60.548877 \n",
       "L 70.42985 59.425478 \n",
       "L 74.053226 58.213317 \n",
       "L 77.676603 56.954015 \n",
       "L 81.29998 55.703719 \n",
       "L 84.923356 54.531474 \n",
       "L 88.546733 53.516857 \n",
       "L 92.17011 52.746754 \n",
       "L 95.793486 52.311212 \n",
       "L 99.416863 52.298486 \n",
       "L 103.040239 52.789406 \n",
       "L 106.663616 53.851555 \n",
       "L 110.286993 55.533704 \n",
       "L 113.910369 57.861215 \n",
       "L 117.533746 60.833015 \n",
       "L 121.157123 64.420547 \n",
       "L 124.780499 68.568987 \n",
       "L 128.403876 73.200663 \n",
       "L 132.027252 78.22024 \n",
       "L 135.650629 83.521129 \n",
       "L 139.274006 88.992272 \n",
       "L 142.897382 94.524569 \n",
       "L 146.520759 100.0162 \n",
       "L 150.144136 105.376417 \n",
       "L 153.767512 110.527535 \n",
       "L 157.390889 115.405204 \n",
       "L 161.014265 119.95721 \n",
       "L 164.637642 124.141356 \n",
       "L 168.261019 127.922829 \n",
       "L 171.884395 131.271705 \n",
       "L 175.507772 134.161002 \n",
       "L 179.131149 136.565573 \n",
       "L 182.754525 138.462091 \n",
       "L 186.377902 139.830048 \n",
       "L 190.001278 140.653763 \n",
       "L 193.624655 140.925108 \n",
       "L 197.248032 140.646678 \n",
       "L 200.871408 139.835025 \n",
       "L 204.494785 138.523533 \n",
       "L 208.118162 136.764449 \n",
       "L 211.741538 134.629546 \n",
       "L 215.364915 132.208909 \n",
       "\" clip-path=\"url(#p0ebccbb441)\" style=\"fill: none; stroke: #d62728; stroke-opacity: 0.5; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_15\">\n",
       "    <path d=\"M 37.81946 92.748331 \n",
       "L 41.442837 90.639595 \n",
       "L 45.066213 88.256332 \n",
       "L 48.68959 85.70597 \n",
       "L 52.312967 83.075493 \n",
       "L 55.936343 80.421471 \n",
       "L 59.55972 77.767011 \n",
       "L 63.183097 75.105258 \n",
       "L 66.806473 72.407989 \n",
       "L 70.42985 69.637044 \n",
       "L 74.053226 66.755936 \n",
       "L 77.676603 63.739399 \n",
       "L 81.29998 60.579162 \n",
       "L 84.923356 57.285298 \n",
       "L 88.546733 53.883517 \n",
       "L 92.17011 50.409618 \n",
       "L 95.793486 46.902883 \n",
       "L 99.416863 43.400234 \n",
       "L 103.040239 39.932725 \n",
       "L 106.663616 36.5252 \n",
       "L 110.286993 33.199119 \n",
       "L 113.910369 29.977846 \n",
       "L 117.533746 26.892989 \n",
       "L 121.157123 23.99018 \n",
       "L 124.780499 21.332679 \n",
       "L 128.403876 19.001787 \n",
       "L 132.027252 17.093477 \n",
       "L 135.650629 15.711622 \n",
       "L 139.274006 14.958642 \n",
       "L 142.897382 14.925108 \n",
       "L 146.520759 15.67981 \n",
       "L 150.144136 17.261859 \n",
       "L 153.767512 19.675913 \n",
       "L 157.390889 22.891103 \n",
       "L 161.014265 26.843612 \n",
       "L 164.637642 31.44231 \n",
       "L 168.261019 36.576362 \n",
       "L 171.884395 42.123726 \n",
       "L 175.507772 47.959258 \n",
       "L 179.131149 53.961577 \n",
       "L 182.754525 60.01809 \n",
       "L 186.377902 66.028028 \n",
       "L 190.001278 71.903756 \n",
       "L 193.624655 77.57083 \n",
       "L 197.248032 82.967346 \n",
       "L 200.871408 88.04323 \n",
       "L 204.494785 92.759757 \n",
       "L 208.118162 97.089445 \n",
       "L 211.741538 101.016149 \n",
       "L 215.364915 104.53494 \n",
       "\" clip-path=\"url(#p0ebccbb441)\" style=\"fill: none; stroke: #9467bd; stroke-opacity: 0.5; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 28.942188 147.225108 \n",
       "L 28.942188 8.625108 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 224.242188 147.225108 \n",
       "L 224.242188 8.625108 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 28.942187 147.225108 \n",
       "L 224.242188 147.225108 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 28.942187 8.625108 \n",
       "L 224.242188 8.625108 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p0ebccbb441\">\n",
       "   <rect x=\"28.942188\" y=\"8.625108\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def rbfkernel(x1, x2, ls=4.):  #@save\n",
    "    dist = distance_matrix(np.expand_dims(x1, 1), np.expand_dims(x2, 1))\n",
    "    return np.exp(-(1. / ls / 2) * (dist ** 2))\n",
    "\n",
    "x_points = np.linspace(0, 5, 50)\n",
    "meanvec = np.zeros(len(x_points))\n",
    "covmat = rbfkernel(x_points,x_points, 1)\n",
    "\n",
    "prior_samples= np.random.multivariate_normal(meanvec, covmat, size=5);\n",
    "d2l.plt.plot(x_points, prior_samples.T, alpha=0.5)\n",
    "d2l.plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70413f3b",
   "metadata": {
    "origin_pos": 7
   },
   "source": [
    "## The Neural Network Kernel\n",
    "\n",
    "Research on Gaussian processes in machine learning was triggered by research on neural networks. Radford Neal was pursuing ever larger Bayesian neural networks, ultimately showing in 1994 (later published in 1996, as it was one of the most infamous NeurIPS rejections) that such networks with an infinite number of hidden units become Gaussian processes with particular kernel functions :cite:`neal1996bayesian`. Interest in this derivation has re-surfaced, with ideas like the neural tangent kernel being used to investigate the generalization properties of neural networks :cite:`matthews2018gaussian` :cite:`novak2018bayesian`. We can derive the neural network kernel as follows.\n",
    "\n",
    "Consider a neural network function $f(x)$ with one hidden layer:\n",
    "\n",
    "$$f(x) = b + \\sum_{i=1}^{J} v_i h(x; u_i).$$\n",
    "\n",
    "$b$ is a bias, $v_i$ are the hidden to output weights, $h$ is any bounded hidden unit transfer function, $u_i$ are the input to hidden weights, and $J$ is the number of hidden units. Let $b$ and $v_i$ be independent with zero mean and variances $\\sigma_b^2$ and $\\sigma_v^2/J$, respectively, and let the $u_i$ have independent identical distributions. We can then use the central limit theorem to show that any collection of function values $f(x_1),\\dots,f(x_n)$ has a joint multivariate Gaussian distribution.\n",
    "\n",
    "The mean and covariance function of the corresponding Gaussian process are:\n",
    "\n",
    "$$m(x) = E[f(x)] = 0$$\n",
    "\n",
    "$$k(x,x') = \\textrm{cov}[f(x),f(x')] = E[f(x)f(x')] = \\sigma_b^2 + \\frac{1}{J} \\sum_{i=1}^{J} \\sigma_v^2 E[h_i(x; u_i)h_i(x'; u_i)]$$\n",
    "\n",
    "In some cases, we can essentially evaluate this covariance function in closed form. Let $h(x; u) = \\textrm{erf}(u_0 + \\sum_{j=1}^{P} u_j x_j)$, where $\\textrm{erf}(z) = \\frac{2}{\\sqrt{\\pi}} \\int_{0}^{z} e^{-t^2} dt$, and $u \\sim \\mathcal{N}(0,\\Sigma)$. Then $k(x,x') = \\frac{2}{\\pi} \\textrm{sin}(\\frac{2 \\tilde{x}^{\\top} \\Sigma \\tilde{x}'}{\\sqrt{(1 + 2 \\tilde{x}^{\\top} \\Sigma \\tilde{x})(1 + 2 \\tilde{x}'^{\\top} \\Sigma \\tilde{x}')}})$.\n",
    "\n",
    "The RBF kernel is _stationary_, meaning that it is _translation invariant_, and therefore can be written as a function of $\\tau = x-x'$. Intuitively, stationarity means that the high-level properties of the function, such as rate of variation, do not change as we move in input space. The neural network kernel, however, is _non-stationary_. Below, we show sample functions from a Gaussian process with this kernel. We can see that the function looks qualitatively different near the origin.\n",
    "\n",
    "## Summary\n",
    "\n",
    "\n",
    "The first step in performing Bayesian inference involves specifying a prior. Gaussian processes can be used to specify a whole prior over functions. Starting from a traditional \"weight space\" view of modelling, we can induce a prior over functions by starting with the functional form of a model, and introducing a distribution over its parameters. We can alternatively specify a prior distribution directly in function space, with properties controlled by a kernel. The function-space approach has many advantages. We can build models that actually correspond to an infinite number of parameters, but use a finite amount of computation! Moreover, while these models have a great amount of flexibility, they also make strong assumptions about what types of functions are a priori likely, leading to relatively good generalization on small datasets.\n",
    "\n",
    "The assumptions of models in function space are intuitively controlled by kernels, which often encode higher level properties of functions, such as smoothness and periodicity. Many kernels are stationary, meaning that they are translation invariant. Functions drawn from a Gaussian process with a stationary kernel have roughly the same high-level properties (such as rate of variation) regardless of where we look in the input space.\n",
    "\n",
    "Gaussian processes are a relatively general model class, containing many examples of models we are already familiar with, including polynomials, Fourier series, and so on, as long as we have a Gaussian prior over the parameters. They also include neural networks with an infinite number of parameters, even without Gaussian distributions over the parameters. This connection, discovered by Radford Neal, triggered machine learning researchers to move away from neural networks, and towards Gaussian processes.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Draw sample prior functions from a GP with an Ornstein-Uhlenbeck (OU) kernel, $k_{\\textrm{OU}}(x,x') = \\exp\\left(-\\frac{1}{2\\ell}||x - x'|\\right)$. If you fix the lengthscale $\\ell$ to be the same, how do these functions look different than sample functions from a GP with an RBF kernel?\n",
    "\n",
    "2. How does changing the _amplitude_ $a^2$ of the RBF kernel affect the distribution over functions?\n",
    "\n",
    "3. Suppose we form $u(x) = f(x) + 2g(x)$, where $f(x) \\sim \\mathcal{GP}(m_1,k_1)$ and $g(x) \\sim \\mathcal{GP}(m_2,k_2)$. Is $u(x)$ a Gaussian process, and if so, what is its mean and covariance function?\n",
    "\n",
    "4. Suppose we form $g(x) = a(x)f(x)$, where $f(x) \\sim \\mathcal{GP}(0,k)$ and $a(x) = x^2$. Is $g(x)$ a Gaussian process, and if so, what is its mean and covariance function? What is the effect of $a(x)$? What do sample functions drawn from $g(x)$ look like?\n",
    "\n",
    "5. Suppose we form $u(x) = f(x)g(x)$, where $f(x) \\sim \\mathcal{GP}(m_1,k_1)$ and $g(x) \\sim \\mathcal{GP}(m_2,k_2)$. Is $u(x)$ a Gaussian process, and if so, what is its mean and covariance function?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a72c6604",
   "metadata": {
    "origin_pos": 8,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/12116)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "required_libs": []
 },
 "nbformat": 4,
 "nbformat_minor": 5
}